# Self-supervised pre-training

RNAErnie is a model for self-supervised pre-training based on non-coding RNA sequences. It uses a multi-stage masked language modeling objective to provide powerful feature representation capabilities for RNA research.

ProphetNet is a sequence-to-sequence pre-trained language model that employs a self-supervised objective of future n-gram prediction, capable of predicting more future tokens

A pre-trained Conformer encoder model based on self-supervised learning, supporting automatic speech recognition tasks for 22 scheduled Indian languages.

Large-scale general-purpose audio encoder trained via self-supervised learning, capable of processing multi-domain audio information including speech, music, and environmental sounds

GPT-2 is a self-supervised pre-trained language model based on the Transformer architecture, which excels at text generation tasks.

RegNetY-128GF feature extraction model, pre-trained using the SEER method on 2 billion random web images with self-supervised learning

ConvNeXt-V2 self-supervised feature representation model, pre-trained using the Fully Convolutional Masked Autoencoder (FCMAE) framework, suitable for image classification and feature extraction tasks.

A self-supervised feature representation model based on ConvNeXt-V2, utilizing the Fully Convolutional Masked Autoencoder (FCMAE) framework for pre-training, suitable for image classification and feature extraction tasks.

This Vision Transformer model is pre-trained using the MSN method and excels in few-shot scenarios, suitable for tasks like image classification

Swin Transformer v2 is a vision Transformer model that achieves efficient image processing through hierarchical feature maps and local window self-attention mechanisms.

This model is a Wav2Vec2 base model pre-trained on 3,000 hours of Vietnamese speech data, suitable for Vietnamese speech recognition tasks, and requires fine-tuning on downstream tasks for use.

RegNet model trained on imagenet-1k, pre-trained in a self-supervised manner on billions of random web images before fine-tuning

XLM-Align is a pre-trained cross-lingual model supporting 94 languages, improving pre-trained cross-lingual models through self-labeled word alignment.

BEiT is a vision Transformer-based image classification model, pre-trained in a self-supervised manner on ImageNet-21k and fine-tuned on ImageNet-1k.

The Wav2Vec2 large model developed by Facebook, pre-trained and fine-tuned on 960 hours of Libri-Light and Librispeech audio data, using self-training objectives, achieving SOTA results on the LibriSpeech test set.

BEiT is an image segmentation model based on Vision Transformer architecture, achieving efficient semantic segmentation through self-supervised pre-training and fine-tuning on the ADE20k dataset.

BEiT is a vision model based on image transformers, employing a BERT-like self-supervised pre-training method. It is first pre-trained and fine-tuned on ImageNet-22k, then further fine-tuned on ImageNet-1k.

BEiT is a Vision Transformer-based model pre-trained on ImageNet-21k through self-supervised learning and fine-tuned on ImageNet-1k for image classification tasks.

ProtT5-XL-BFD is a self-supervised pre-trained model based on protein sequences, using the T5 architecture, trained on 2.1 billion protein sequences for protein feature extraction and downstream task fine-tuning.

A protein sequence pre-training model based on T5-3B architecture that captures biophysical properties through self-supervised learning